Tag: communication overlap
2 reviews
Kernel optimization for LLM training: techniques, trade-offs, and use cases
Which kernel-level optimization techniques most improve LLM training efficiency, and what are the trade-offs and use cases of each?
Kernel-level optimization is where most LLM training efficiency gains of the last six years come from, and the field splits into five families: fused attention kernels, low-precision GEMM kernels, mixture-of-experts kernels, communication-overlap kernels, and fused elementwise/optimizer kernels. The evidence is consistent that each family delivers large, workload-specific gains — 2x on attention, 25-75% on FP8 training, 1.4-5.7x on MoE layers, 1.2-2.9x on long-context training — but the gains are conditional on hardware generation, model architecture, and engineering effort, and almost every headline number is vendor- or single-group-benchmarked. The main caveat: no shared benchmark exists, so cross-paper 'x-times faster' claims are not directly comparable.
Optimizing MLPerf Training closed-division submissions: what the literature shows
Which methods documented in the literature most reduce time-to-quality for MLPerf Training closed-division submissions, given the rule that model, optimizer and most hyperparameters are fixed?
A review of 47 sources on how to minimize time-to-quality in the MLPerf Training closed division, where the model, optimizer, and most hyperparameters are fixed by the rules. The levers the evidence supports are: picking the global batch size that sits at the reference-convergence boundary, reduced-precision training (bf16/fp8, now also fp4), distributed-parallelism and communication-overlap engineering, data-pipeline and kernel work, and benchmark-specific plays for DLRM, LoRA fine-tuning, and MoE pretraining. Each lever is worth roughly 1.2–2× in isolation and they compound, but the convergence-speed cap (Reference Convergence Points) means the winning move is maximum throughput at the RCP boundary, not faster convergence. Confidence is moderate: the per-lever results are replicated systems literature, but no public study directly tests the full closed-division strategy.